Exploring and exploiting the seemingly innocent theorem behind Double Machine Learning. The theorem, rooted in econometrics, states that if we have a linear model that predicts an outcome variable based on multiple features, and we want to understand the causal effect of a specific feature on the outcome, we can use the residuals of the model as an instrumental variable to estimate the causal effect.
Discusses reasons why clustering in data science might not produce desired results and how to address these issues.
This article features a curated list of the top data science articles published in July, covering topics such as LLM apps, chatGPT, data visualization, multi-agent AI systems, and essential data science skills for 2024.
An article discussing the current state, recent approaches, and future directions of prompt engineering in data and machine learning. It includes several links to relevant articles and tutorials on the topic.
An overview of the LIDA library, including how to get started, examples, and considerations going forward, with a focus on large language models (LLMs) and image generation models (IGMs) in data visualization and business intelligence.
This article discusses the importance of understanding and memorizing classification metrics in machine learning. The author shares their own experience and strategies for memorizing metrics such as accuracy, precision, recall, F1 score, and ROC AUC.
This article explains the PCA algorithm and its implementation in Python. It covers key concepts such as Dimensionality Reduction, eigenvectors, and eigenvalues. The tutorial aims to provide a solid understanding of the algorithm's inner workings and its application for dealing with high-dimensional data and the curse of dimensionality.
This tutorial covers fine-tuning BERT for sentiment analysis using Hugging Face Transformers. Learn to prepare data, set up environment, train and evaluate the model, and make predictions.
This article explains the concept and use of Friedman's H-statistic for finding interactions in machine learning models.
- The H-stat is a non-parametric method that works well with ordinal variables, and it's useful when the interaction is not linear.
- The H-stat compares the average rank of the response variable for each level of the predictor variable, considering all possible pairs of levels.
- The H-stat calculates the sum of these rank differences and normalizes it by the total number of observations and the number of levels in the predictor variable.
- The lower the H-stat, the stronger the interaction effect.
- The article provides a step-by-step process for calculating the H-stat, using an example with a hypothetical dataset about the effects of asbestos exposure on lung cancer for smokers and non-smokers.
- The author also discusses the assumptions of the H-stat and its limitations, such as the need for balanced data and the inability to detect interactions between more than two variables.
- The H-stat is a non-parametric method that works well with ordinal variables, and it's useful when the interaction is not linear.
- The H-stat compares the average rank of the response variable for each level of the predictor variable, considering all possible pairs of levels.
- The H-stat calculates the sum of these rank differences and normalizes it by the total number of observations and the number of levels in the predictor variable.
- The lower the H-stat, the stronger the interaction effect.
- The article provides a step-by-step process for calculating the H-stat, using an example with a hypothetical dataset about the effects of asbestos exposure on lung cancer for smokers and non-smokers.
- The author also discusses the assumptions of the H-stat and its limitations, such as the need for balanced data and the inability to detect interactions between more than two variables.
The Towards Data Science team highlights recent articles on the rise of open-source LLMs, ethical considerations with chatbots, potential manipulation of LLM recommendations, and techniques for temperature scaling and re-ranking in generative AI.
